linkExtractor

Function for extracting URLs for links found on crawled pages.

By default, Algolia queues all URLs that comply with the pathsToMatch and fileTypesToMatch actions, and the exclusionPatterns parameter. You can override this default logic by providing a linkExtractor function that overrides this default logic and returns its own list of URLs to queue.

Parameters

$ · object

A Cheerio instance with the HTML of the crawled page. For more information, see Extracting data with Cheerio.

defaultExtractor · function

A Cheerio instance with the HTML of the crawled page. The Crawler’s default URL discovery function It returns an array of strings, each representing a URL on the page that matches the crawler’s configuration.

url · URL

URL of the page that was just crawled.

Examples

JavaScript

{
    linkExtractor: ({ $, url, defaultExtractor }) => {
    if (/example.com\/doc\//.test(url.href)) {
        // For all pages under /doc, only queue the first found link
        return defaultExtractor().slice(0,1);
    }
    // Otherwise, use the default logic (queue all found links)
    return defaultExtractor();
    },
}

JavaScript

{
linkExtractor: ({ $, url, defaultExtractor }) => {
    // This turns off link discovery, except for URLs listed in sitemap.xml
    return /sitemap.xml/.test(url.href) ? defaultExtractor() : [];
},
}

JavaScript

{
linkExtractor: ({ $ }) => {
    // Access the DOM and extract what you specify
    return [$('.my-link').attr('href')]
},
}